Activation of extended reality actuators based on content analysis

ABSTRACT

In one example, a method performed by a processing system in a telecommunications network includes acquiring the media stream and identifying an anchor in a scene of the media stream. The anchor is a presence in the scene that has a physical effect on the scene. A type and a magnitude of the physical effect of the anchor on the scene is estimated. An actuator in a vicinity of the user endpoint device that is capable of producing a physical effect in the real world to match the physical effect of the anchor on the scene is identified. A signal is sent to the actuator. The signal controls the actuator to produce the physical effect in the real world when the physical effect of the anchor on the scene occurs in the media stream.

This application is a continuation of U.S. patent application Ser. No.16/682,379, filed on Nov. 13, 2019, now U.S. Pat. No. 11,282,281, whichis herein incorporated by reference in its entirety.

The present disclosure relates generally to extended reality (XR) media,and relates more particularly to devices, non-transitorycomputer-readable media, and methods for activating extended realityactuators based on content analysis.

BACKGROUND

Extended reality (XR) is an umbrella term used to describe various typesof immersive technology, including augmented reality (AR), virtualreality (VR), and mixed reality (MR), in which the real-worldenvironment may be enhanced or augmented with virtual,computer-generated objects or actions. As XR technology improves, therehas been an increasing effort to make the user experience morerealistic. For instance, in addition to inserting visible and/or audiblevirtual objects into the presentation of an XR media, the presentationmay also include manipulation of physical objects in the proximity ofthe user, e.g., by activating actuators that introduce physical motion,tactile effects, temperature changes, and the like.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present disclosure can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 illustrates an example network related to the present disclosure;

FIG. 2 illustrates a flowchart of a method 200 for activating actuatorsbased on content analysis of the media in accordance with the presentdisclosure; and

FIG. 3 depicts a high-level block diagram of a computing devicespecifically programmed to perform the functions described herein.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION

In one example, the present disclosure enhances extended reality (XR)media by activating actuators based on content analysis of the media,where the content analysis may potentially be performed in real time(e.g., as the media is being delivered to a user endpoint device). Inone example, a method formed by a processing system in atelecommunications network includes acquiring a media stream that isbeing delivered to a user endpoint device in the telecommunicationsnetwork, identifying an anchor in a scene of the media stream, whereinthe anchor comprises a presence in the scene that has a physical effecton the scene, estimating a type and a magnitude of the physical effectof the anchor on the scene, identifying an actuator in a vicinity of theuser endpoint device that is capable of producing a physical effect inthe real world to match the physical effect of the anchor on the scene,and sending a signal to the actuator, wherein the signal controls theactuator to produce the physical effect in the real world when thephysical effect of the anchor on the scene occurs in the media stream.

In another example, a non-transitory computer-readable medium storesinstructions which, when executed by a processing system in atelecommunications network, cause the processing system to performoperations. The operations include acquiring a media stream that isbeing delivered to a user endpoint device in the telecommunicationsnetwork, identifying an anchor in a scene of the media stream, whereinthe anchor comprises a presence in the scene that has a physical effecton the scene, estimating a type and a magnitude of the physical effectof the anchor on the scene, identifying an actuator in a vicinity of theuser endpoint device that is capable of producing a physical effect inthe real world to match the physical effect of the anchor on the scene,and sending a signal to the actuator, wherein the signal controls theactuator to produce the physical effect in the real world when thephysical effect of the anchor on the scene occurs in the media stream.

In another example, a device includes a processor and a non-transitorycomputer-readable medium that stores instructions which, when executedby the processor, cause the processing system to perform operations. Theoperations include acquiring a media stream that is being delivered to auser endpoint device in a telecommunications network, identifying ananchor in a scene of the media stream, wherein the anchor comprises apresence in the scene that has a physical effect on the scene,estimating a type and a magnitude of the physical effect of the anchoron the scene, identifying an actuator in a vicinity of the user endpointdevice that is capable of producing a physical effect in the real worldto match the physical effect of the anchor on the scene, and sending asignal to the actuator, wherein the signal controls the actuator toproduce the physical effect in the real world when the physical effectof the anchor on the scene occurs in the media stream.

As discussed above, as XR technology improves, there has been anincreasing effort to make the user experience more realistic. Forinstance, in addition to inserting visible and/or audible virtualobjects into the presentation of an XR media, the presentation may alsoinclude manipulation of physical objects in the proximity of the user,e.g., by activating actuators that introduce physical motion, tactileeffects, temperature changes, and the like.

In some cases, the stream of XR media may include metadata thatindicates, to a controller, when and how to activate specific actuatorsor types of actuators. The metadata may be embedded in one or moreadditional synchronized tracks of the XR media. For instance, a videochunk downloaded from a server may include metadata that specifies atime at which to activate a rumble mechanism in a chair or at which todim the lights. However, coordinating the activation of the actuatorswith precise events in the media stream is an expensive, and typicallymanually driven, process. For instance, a human operator must review themedia stream and manually add the proper metadata at the appropriatetimes. Moreover, because this process is typically performed in advanceof the media stream being presented on an endpoint device, it isdifficult to adapt effectively for live events or for media streams forwhich no metadata has been inserted in advance.

Examples of the present disclosure perform content analysis of an XRmedia stream e.g., potentially in real time, as the XR media stream isbeing presented) in order to detect time points in the media stream atwhich a physical actuator may be actuated to enhance the presentation.In particular, the physical actuator may produce physical effects in thereal world that mimic the physical effects of one or more objects oreffects in a scene of the media stream, thereby providing an experiencethat is more immersive. Further examples of the present disclosure maygenerate, at the identified time points, signals that may be transmittedto the actuators (e.g., either directly or via a centralized controller)in order to activate the actuators and produce a manipulation of aphysical object in the user's vicinity (e.g., produce a physical effectthat can be felt by the user). Further examples still may adapt theactivation of the actuators for the specific user to whom the XR mediais being presented, in order to personalize the immersive experience.

Although examples of the present disclosure may be discussed asoperating in real time or near-real time (e.g., as a media stream isbeing delivered to a user endpoint device), which is useful whenstreaming live events or media streams for which no metadata has beeninserted in advance, the examples of the present disclosure do notalways operate in real time. For instance, examples of the presentdisclosure may also be applied (in less than real time) to enhancepre-produced media streams as a more cost effective (and potentiallyfaster and more accurate) alternative to manual processing.

To better understand the present disclosure, FIG. 1 illustrates anexample network 100, related to the present disclosure. As shown in FIG.1, the network 100 connects mobile devices 157A, 157B, 167A and 167B,and home network devices such as home gateway 161, set-top boxes (STBs)162A, and 162B, television (TV) 163A and TV 163B, home phone 164, router165, personal computer (PC) 166, and so forth, with one another and withvarious other devices via a core network 110, a wireless access network150 (e.g., a cellular network), an access network 120, other networks140 and/or the Internet 145.

In one example, wireless access network 150 comprises a radio accessnetwork implementing such technologies as: global system for mobilecommunication (GSM), e.g., a base station subsystem (BSS), or IS-95, auniversal mobile telecommunications system (UMTS) network employingwideband code division multiple access (WCDMA), or a CDMA3000 network,among others. In other words, wireless access network 150 may comprisean access network in accordance with any “second generation” (2G),“third generation” (3G), “fourth generation” (4G), Long Term Evolution(LTE) or any other yet to be developed future wireless/cellular networktechnology including “fifth generation” (5G) and further generations.While the present disclosure is not limited to any particular type ofwireless access network, in the illustrative example, wireless accessnetwork 150 is shown as a UMTS terrestrial radio access network (UTRAN)subsystem. Thus, elements 152 and 153 may each comprise a Node B orevolved Node B (eNodeB).

In one example, each of mobile devices 157A, 157B, 167A, and 167B maycomprise any subscriber/customer endpoint device configured for wirelesscommunication such as a laptop computer, a Wi-Fi device, a PersonalDigital Assistant (PDA), a mobile phone, a smartphone, an email device,a computing tablet, a messaging device, a wearable smart device (e.g., asmart watch or fitness tracker), a gaming console, and the like. In oneexample, any one or more of mobile devices 157A, 157B, 167A, and 167Bmay have both cellular and non-cellular access capabilities and mayfurther have wired communication and networking capabilities.

As illustrated in FIG. 1, network 100 includes a core network 110. Inone example, core network 110 may combine core network components of acellular network with components of a triple play service network; wheretriple play services include telephone services, Internet services andtelevision services to subscribers. For example, core network 110 mayfunctionally comprise a fixed mobile convergence (FMC) network, e.g., anIP Multimedia Subsystem (IMS) network. In addition, core network 110 mayfunctionally comprise a telephony network, e.g., an InternetProtocol/Multi-Protocol Label Switching (IP/MPLS) backbone networkutilizing Session Initiation Protocol (SIP) for circuit-switched andVoice over Internet Protocol (VoIP) telephony services. Core network 110may also further comprise a broadcast television network, e.g., atraditional cable provider network or an Internet Protocol Television(IPTV) network, as well as an Internet Service Provider (ISP) network.The network elements 111A-111D may serve as gateway servers or edgerouters to interconnect the core network 110 with other networks 140(which may include servers 149), Internet 145, wireless access network150, access network 120, and so forth. As shown in FIG. 1, core network110 may also include a plurality of television (TV) servers 112, aplurality of content servers 113, a plurality of application servers114, an advertising server (AS) 117, and an extended reality (XR) server115 (e.g., an application server). For ease of illustration, variousadditional elements of core network 110 are omitted from FIG. 1.

With respect to television service provider functions, core network 110may include one or more television servers 112 for the delivery oftelevision content, e.g., a broadcast server, a cable head-end, and soforth. For example, core network 110 may comprise a video super huboffice, a video hub office and/or a service office/central office. Inthis regard, television servers 112 may interact with content servers113, advertising server 117, and XR server 115 to select which videoprograms, or other content and advertisements to provide to the homenetwork 160 and to others.

In one example, content servers 113 may store scheduled televisionbroadcast content for a number of television channels, video-on-demandprogramming, local programming content, gaming content, and so forth.The content servers 113 may also store other types of media that are notaudio/video in nature, such as audio-only media (e.g., music, audiobooks, podcasts, or the like) or video-only media (e.g., imageslideshows). For example, content providers may upload various contentsto the core network to be distributed to various subscribers.Alternatively, or in addition, content providers may stream variouscontents to the core network for distribution to various subscribers,e.g., for live content, such as news programming, sporting events, andthe like. In one example, advertising server 117 stores a number ofadvertisements that can be selected for presentation to viewers, e.g.,in the home network 160 and at other downstream viewing locations. Forexample, advertisers may upload various advertising content to the corenetwork 110 to be distributed to various viewers.

In one example, XR server 115 may generate computer-generated contentincluding electronic signals that control actuators on various devicesto generate physical effects. When the physical effects are combinedwith a media stream, and potentially with other computer-generatedcontent generated by the XR server 115 such as computer-generatedgraphics and/or audio, an immersive media is produced. For instance, asignal may contain an instruction for a smart thermostat to raise atemperature, for a haptic feedback mechanism to provide a rumble effect,or for a digital scent technology device to emit a manufactured scent.The physical effects may be synchronized with the visual and/or audio ofthe media stream in order to make a user's experience of the mediastream more immersive or engaging.

In one example the XR server 115 may store data provided by users of thenetwork 100. The stored data may be provided directly to the XR server115 by the users, e.g., via the mobile devices 157A, 157B, 167A, and167B, the PC 166, the home phone 164, the TVs 163A and 163B, and/orInternet of Things (IoT) devices 168A and 168B. Alternatively, the datamay be retrieved from network storage, e.g., application servers 114, bythe XR server 115. For instance the stored data may comprise userprofiles maintained by a network service (e.g., an Internet serviceprovider, a streaming media service, a gaming subscription, etc.),portions of social media profiles maintained by a social media web site(e.g., a social networking site, a blogging site, a photo-sharing site,etc.), or the like. The data may indicate information about the users,such as the users' interests (e.g., favorite genres of movies ortelevision shows, favorite games, etc.), ages, devices (e.g., mobiledevices, IoT devices, gaming devices, etc.), medical or other conditionsthat may affect the extent and the types of the physical effects towhich the user may be exposed, and the like.

In one example, any or all of the television servers 112, contentservers 113, application servers 114, XR server 115, and advertisingserver 117 may comprise a computing system, such as computing system 300depicted in FIG. 3.

In one example, the access network 120 may comprise a Digital SubscriberLine (DSL) network, a broadband cable access network, a Local AreaNetwork (LAN), a cellular or wireless access network, a 3^(rd) partynetwork, and the like. For example, the operator of core network 110 mayprovide a cable television service, an IPTV service, or any other typeof television service to subscribers via access network 120. In thisregard, access network 120 may include a node 122, e.g., a mini-fibernode (MFN), a video-ready access device (VRAD) or the like. However, inanother example node 122 may be omitted, e.g., for fiber-to-the-premises(FTTP) installations. Access network 120 may also transmit and receivecommunications between home network 160 and core network 110 relating tovoice telephone calls, communications with web servers via the Internet145 and/or other networks 140, and so forth.

Alternatively, or in addition, the network 100 may provide televisionservices to home network 160 via satellite broadcast. For instance,ground station 130 may receive television content from televisionservers 112 for uplink transmission to satellite 135. Accordingly,satellite 135 may receive television content from ground station 130 andmay broadcast the television content to satellite receiver 139, e.g., asatellite link terrestrial antenna (including satellite dishes andantennas for downlink communications, or for both downlink and uplinkcommunications), as well as to satellite receivers of other subscriberswithin a coverage area of satellite 135. In one example, satellite 135may be controlled and/or operated by a same network service provider asthe core network 110. In another example, satellite 135 may becontrolled and/or operated by a different entity and may carrytelevision broadcast signals on behalf of the core network 110.

In one example, home network 160 may include a home gateway 161, whichreceives data/communications associated with different types of media,e.g., television, phone, and Internet, and separates thesecommunications for the appropriate devices. The data/communications maybe received via access network 120 and/or via satellite receiver 139,for instance. In one example, television data is forwarded to set-topboxes (STBs)/digital video recorders (DVRs) 162A and 162B to be decoded,recorded, and/or forwarded to television (TV) 163A and TV 163B forpresentation. Similarly, telephone data is sent to and received fromhome phone 164; Internet communications are sent to and received fromrouter 165, which may be capable of both wired and/or wirelesscommunication. In turn, router 165 receives data from and sends data tothe appropriate devices, e.g., personal computer (PC) 166, mobiledevices 167A and 167B, and so forth. In one example, router 165 mayfurther communicate with TV (broadly a display) 163A and/or 163B, e.g.,where one or both of the televisions is a smart TV. In one example,router 165 may comprise a wired Ethernet router and/or an Institute forElectrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi) router, andmay communicate with respective devices in home network 160 via wiredand/or wireless connections.

IoT devices 168A and 168B may include any types of devices that arecapable of being controlled automatically and/or remotely. For instance,the IoT devices 168A and 168B may include “smart” home devices, such asa smart thermostat, a smart lighting system, or the like. The IoTdevices 168A and 168B may also include gaming devices, such as gamingcontrollers, a gaming chair, or the like. Although FIG. 1 illustratestwo IoT devices 168A and 168B, it will be appreciated that the homenetwork 160 may include any number of IoT devices. A greater number andvariety of IoT devices may allow for a more immersive experience to begenerated.

Those skilled in the art will realize that the network 100 may beimplemented in a different form than that which is illustrated in FIG.1, or may be expanded by including additional endpoint devices, accessnetworks, network elements, application servers, etc. without alteringthe scope of the present disclosure. For example, core network 110 isnot limited to an IMS network. Wireless access network 150 is notlimited to a UMTS/UTRAN configuration. Similarly, the present disclosureis not limited to an IP/MPLS network for VoIP telephony services, or anyparticular type of broadcast television network for providing televisionservices, and so forth.

To further aid in understanding the present disclosure, FIG. 2illustrates a flowchart of a method 200 for activating actuators basedon content analysis of the media in accordance with the presentdisclosure. In one example, the method 200 may be performed by an XRserver that is configured to generate computer-generated content thatmay be blended or synchronized with a live or pre-recorded media toproduce an immersive media, such as the XR server 115 illustrated inFIG. 1. However, in other examples, the method 200 may be performed byanother device, such as the processor 302 of the system 300 illustratedin FIG. 3. For the sake of example, the method 200 is described as beingperformed by a processing system.

The method 200 begins in step 202. In step 204, the processing systemmay acquire a media stream that is being delivered to a user endpointdevice. In one example, the media stream may comprise a media stream ofa live event (e.g., a sporting event or video that is being capturedlive by the user endpoint device). However, in another example, themedia stream may comprise a pre-recorded media stream (e.g., a film, atelevision show, a video game, or the like). The user endpoint devicemay be any device that is capable of presenting at least a portion of animmersive media, such as a television, a mobile phone, a head mounteddisplay or other wearable display device, an Internet enabled radio orportable music device, and/or the like. In some examples, the userendpoint device may include or be in communication with a device thatprovides some sort of tactile or haptic feedback (e.g., buzzing,rumbling, vibrations, and/or the like). In some examples, the userendpoint device may be in communication with one or more IoT devicesthat are in close proximity to the user endpoint device, such as a smartthermostat, a smart lighting system, a smart speaker system, and/or thelike.

In step 206, the processing system may identify one or more anchors in ascene of the media stream (e.g., a scene that has not yet been streamedto the user endpoint device). In one example, an anchor is a presence inthe scene (e.g., an object or an environmental effect appearing in thescene) that has a physical effect on the scene. An anchor may thereforepotentially serve as a source of a physical effect in the real world(e.g., a physical motion, a tactile effect, a temperature change, and/orthe like). The physical effect may be an effect that can be produced byan actuator that is located within some proximity (e.g., within x feetof) of the user endpoint device. For instance, the physical effect couldbe produced by the user endpoint device, by an IoT device, by a seat orother piece of furniture with an integrated haptic feedback mechanism(e.g., a gaming chair), or by any other device that is capable ofproducing a physical effect.

In one example, the anchor may be an object (e.g., a person or a thing)that appears in the scene and that interacts with some other object inthe scene. For instance, the anchor object may collide with the otherobject, may push or pull the other object or be pushed or pulled by theother object, or the like). As an example, the anchor may be a skier whohas mounted a body camera to himself. The skier may move in a particularway as he skis down the mountain, and may experience different physicaleffects as a result of interaction with other skiers and objects (trees,moguls, terrain park features, etc.).

In another example, the anchor may be an environmental effect of thescene on the camera(s) that filmed the scene. For instance, a camerafilming the scene may shake due to environmental effects (e.g., wind),due to interaction with objects (e.g., a vehicle passing quickly inclose proximity), due to shaking of a human operator's hand, and/or thelike.

In another example, the anchor may be an event that occurs within thescene, where the event has some physical effect on the scene. Forinstance, the event may be an explosion, a weather effect (e.g., rain,snow, earthquake).

In step 208, the processing system may estimate the type and/or themagnitude of a physical effect of at least one of the anchors on thescene. For instance, the processing system may use image processingtechniques, audio processing techniques, and/or other video analysistechniques on components of the media stream to estimate the magnitudeof a force acting on an anchor, or the force generated by the anchoracting on an object. As an example, if the anchor collides with anotherobject, the force of the collision may be estimated based on estimatesof the mass of the anchor, the acceleration of the anchor prior to thecollision, the mass of the other object, the acceleration of the otherobject prior to the collision, the distances traveled by the anchorand/or the other object after the collision, or the like. The masses,accelerations, and distances may be estimated using image processingtechniques.

Similarly, if the anchor is an event, such as an explosion, the force ofthe explosion and/or the temperature of the explosion may be estimatedbased on the size of the explosion, the distance between the explosionand the camera filming the explosion, and/or other quantifiablecharacteristics that can be estimated based on image processingtechniques. If the anchor is a baseball that has just been hit, audioprocessing techniques may be able to estimate a force with which the bathit the ball based on the sound of the ball coming off of the bat. Thus,analysis of the various components of the media stream (e.g., visualcomponent, audio component, and/or other components) may allow thephysical effects of an anchor on a scene of the media stream to bequantified.

In some examples, if the anchor is rendered using special effects (e.g.,in a computer generated media), metadata associated with the mediastream may include explicit force analyses.

In step 210, the processing system may identify one or more actuators onor in the vicinity of the user endpoint device that may be used toproduce a physical effect in the real world that matches one or more ofthe estimated physical effects estimated in step 208. For instance, ifthe anchor comprises an explosion, the estimated physical effects mayinclude a rumbling of a certain magnitude and a sudden surge intemperature up to a certain maximum temperature. The rumbling effect maybe produced by a rumble mechanism in the user endpoint device or in agame controller that is communicatively coupled to the user endpointdevice. Alternatively, if the user is sitting in a gaming chair or on asimilar type of furniture while viewing the media stream, the rumblingeffect may be produced by a rumble mechanism in the chair. Thetemperature effect may be produced by a smart thermostat that controlsthe ambient temperature of the location in which the user is viewing themedia stream.

In step 212, the processing system may select one or more physicaleffects to present to the user and one or more actuators (e.g.,actuators on or in the vicinity of the user endpoint device) to producethe physical effects. In one example, the selection in step 212 ispersonalized for the user. For instance, the user (and/or the user'sdevices, including the user endpoint device, IoT devices, and the like)may be registered with a service that provides immersive content, andthe service may maintain a database that stores a profile for the user.The profile may include information about the user includingpreferences, interests, medical conditions, restrictions on contentratings, and the like. Thus, the profile may be used to create a uniqueimmersive experience that is tailored to the user. For instance, if theprofile indicates that the user is prone to seizures, the processingsystem may refrain from selecting physical effects that include thestrobing or flashing of bright lights. Similarly, if the user hasindicated that he dislikes physical effects that involve temperaturechanges, the processing system may refrain from selecting physicaleffects that involve manipulation of a smart thermostat. Thus, the typeand/or magnitude of the physical effects presented to the user may betailored to minimize aggravation of medical conditions, to matchselected preferences or interests, to respect content restrictions(e.g., which may be put in place by parents), and the like.

In one example, selection of the physical effects and actuators may alsotake into account the known capabilities of the user endpoint deviceand/or devices within the vicinity of the user endpoint device. Forinstance, the user profile may additional store information about thedevice used by the user when experiencing immersive media (e.g.,including the user endpoint device and/or the devices within thevicinity of the user endpoint device). The information stored for anyone of these devices may include, for example, device type (e.g., mobilephone, gaming controller, smart thermostat, etc.), device identifier(e.g., IP address, media access control address, etc.), and devicecapabilities (e.g., includes rumble mechanism, can operate withinpredefined temperature range, etc.). In one example, the devicecapabilities may further indicate the current status of a device's moreephemeral capabilities (e.g., current battery life, current connectivitystrength, etc.). Thus, the profile may be used to determine whichphysical effects can be feasibly produced for the user based on thedevices available to the user at the time. For instance, if the profileindicates that the user does not have access to a smart thermostat, thenthe processing system may disregard any physical effects that requirethe raising or lowering of the ambient temperature. Similarly, if aphysical effect comprises a low-frequency sound effect that the user'sspeaker system is not capable of producing, then the processing systemmay instead select a physical effect that comprises a high-frequencyvibration to mimic the sound effect. Thus, the type and/or magnitude ofthe physical effects presented to the user may be tailored toaccommodate the capabilities of the devices available to the user.

In step 214, the processing system may send one or more signals to oneor more actuators (or devices including actuators) that are capable ofproducing the physical effects selected in step 212. The signals maycontrol the one of more actuators to produce physical effects in thereal world that match estimated physical effects of the one or moreactuators in the real world. For instance, if a physical effectcomprising the raising of an ambient temperature to simulate a suddenheat surge produced by an explosion, then the processing system may senda signal to a smart thermostat to raise the ambient temperature to aspecified temperature for a specified amount of time. Similarly, if aphysical effect comprises a rumble to simulate the tremors that may becaused by an explosion, then the processing system may send a signal toa rumble mechanism in a gaming chair, a gaming controller, a mobilephone, or the like to generate a rumble of a specified intensity for aspecified amount of time. The signals may indicate a time at which thephysical effects should be produced, so that the physical effect isproperly synchronized with the media being streamed. For instance, thesignal may identify specific frame identifiers of the media or specifictime stamps in the media that allow the actuators to properlysynchronize the physical effects.

In one example, the signals may be sent directly to the devices thatinclude the actuators for producing the physical effects. In otherexamples, however, the signals may be sent to a centralized controller,a router, a home gateway, or the like that extracts intendeddestinations from the signals (e.g., from headers of packets containingthe signals) and then forwards the signals to those intendeddestinations. Upon receipt of the signals, the devices may cache thesignals for use at the designated times.

In optional step 216 (illustrated in phantom), the processing system mayreceive feedback regarding the presentation of the physical effects tothe user. In one example, the feedback may comprise explicit feedback.For instance, the user may provide feedback in the form of a verbalstatement, a text input, a button push, or the like via a device (e.g.,the user endpoint device). Similarly, the user may update his profile toindicate whether or not he liked a specific physical effect. In anotherexample, however, the feedback may be more implicit. For instance,sensors in the user endpoint device and/or in devices in the vicinity ofthe user endpoint device may be used to monitor the user's reaction tothe physical effects. As an example, if the user smiles or cheers, thismay be interpreted as a positive feedback. Alternatively, if the userscreams, or if his heart rate exceeds some predefined acceptablethreshold, this may be interpreted as negative feedback.

In optional step 218 (illustrated in phantom), the processing system mayupdate the user's profile based on the feedback. For instance, theupdate may indicate that the user does or does not like certain types ofphysical effects, or likes certain types of physical effects only whensubject to certain restrictions (e.g., do not raise the temperature pastx degrees Fahrenheit, do not produce a rumbling effect with more than yintensity, etc.). These updates may allow the processing system tobetter tailor the physical effects to the user in the future (e.g., byruling out and/or promoting certain types of physical effects in step212). Thus, future iterations of the method 200 (including futureiterations of step 212) may be adjusted based on the feedback receivedin step 216.

The method 200 may end in step 220. However, it should be noted that themethod may also loop back to step 204. For instance, the processingsystem may continuously iterate through steps 204-218 for as long as theuser is viewing the streaming media, so that the physical effects are aconsistent part of the immersive viewing experience.

Thus, examples of the present disclosure utilize automated contentanalytics to provide an immersive media experience. Moreover, since thesignals used to control the actuators are generated automatically, withlittle or no manual human intervention, the immersive media experiencecan be provided in a more cost-effective manner. The disclosed methodcan also be used to transform a live media stream, or a pre-recordedmedia stream for which no immersive metadata has been pre-programmed,into an immersive stream in real or near-real time (e.g. as the mediastream is being streamed to a user endpoint device). However, in otherexamples, the disclosed method is not performed in real time, but couldbe performed before or after streaming of the media stream. As such, thedisclosed method could even be used to transform user-generated content(such as user-generated videos that have been shared via social media)into an immersive media experience.

It should be noted that any given scene of a media may include more thanone anchor. For instance, if a user is viewing a skiing scene from theperspective of a skier, the skier may comprise one anchor and theweather (e.g., snow) may comprise another anchor. Thus, the user mayfeel a rumble as the skier skis through a set of moguls and may alsofeel a drop in temperature or a gust of wind to mimic the environmentalconditions on the mountain. Moreover, the anchor(s) may change fromscene to scene, or even within a single scene. For instance, referringback to the skiing scene, a first anchor may correspond to a firstskier, and the user may experience the media from the perspective of thefirst skier. However, when the first skier passes a second skier on themountain, the anchor may switch to the second skier, and the user maybegin experiencing the media from the perspective of the second skier.

Examples of the present disclosure may be extended to synchronize theimmersive media experiences of two or more users. For instance, twousers may be watching the same football game. However, the first usermay have a first player designated as an anchor for the immersiveexperience, while the second user may have a second player designated asan anchor for the immersive experience. If the first player were to bepushed on the back by the second player, the first user's experience(getting pushed on the back) could be synchronized with the seconduser's experience (pushing forward on the other player's back).

Further examples of the present disclosure may allow for user-generatedcommunications (e.g., shouts, cheering, feedback from a user'smovements) to be shared across two or more collaborative immersiveexperiences. For instance, if a first user and a second user arewatching a live football game and the first user makes a movement thatis equivalent to stomping on the bleachers, actuators in the vicinity ofthe second user may be controlled to mimic the stomping action (e.g.,such that the second user can “feel” other spectators stomping on thebleachers).

Examples of the present disclosure also allow for the creation ofmultiple different immersive experiences based on the same mediacontent, potentially based on different user preferences. For instance,the experience of a user who prefers to watch a football game from theperspective of the crowd could be tailored to mimic crowd-based actions(e.g., cheering, stomping on the bleachers, etc.), while the experienceof a user who prefers to watch a football game from the perspective of aplayer (e.g., through volumetric video) could be tailored to mimicplayer-based actions (e.g., getting pushed or tackled).

It should also be noted that the personalization of the immersiveexperience need not necessarily be static. For instance, as discussedabove, if a media depicts an explosion, a smart thermostat in thevicinity of the user may raise the ambient temperature so that the user“feels” the heat from the explosion. However, the user may notnecessarily remain in one place while viewing the media. For instance,the user may be walking around. In this case, the smart thermostat maybe controlled to adjust the amount of heat that the user feels based onthe user's movement (e.g., if the user moves closer to the explosion,the ambient temperature may be raised so that the user feels more heat;as the user moves away from the explosion, the ambient temperature maybe lowered so that the user feels less heat). Thus, control of theactuators as discussed herein may be further tied to the user's viewingposition or location, which may not be static.

Although not expressly specified above, one or more steps of the method200 may include a storing, displaying and/or outputting step as requiredfor a particular application. In other words, any data, records, fields,and/or intermediate results discussed in the method can be stored,displayed and/or outputted to another device as required for aparticular application. Furthermore, operations, steps, or blocks inFIG. 2 that recite a determining operation or involve a decision do notnecessarily require that both branches of the determining operation bepracticed. In other words, one of the branches of the determiningoperation can be deemed as an optional step. However, the use of theterm “optional step” is intended to only reflect different variations ofa particular illustrative embodiment and is not intended to indicatethat steps not labelled as optional steps to be deemed to be essentialsteps. Furthermore, operations, steps or blocks of the above describedmethod(s) can be combined, separated, and/or performed in a differentorder from that described above, without departing from the examples ofthe present disclosure.

FIG. 3 depicts a high-level block diagram of a computing devicespecifically programmed to perform the functions described herein. Forexample, any one or more components or devices illustrated in FIG. 1 ordescribed in connection with the method 200 may be implemented as thesystem 300. For instance, a server (such as might be used to perform themethod 200) could be implemented as illustrated in FIG. 3.

As depicted in FIG. 3, the system 300 comprises a hardware processorelement 302, a memory 304, a module 305 for activating actuators basedon content analysis of the media, and various input/output (I/O) devices306.

The hardware processor 302 may comprise, for example, a microprocessor,a central processing unit (CPU), or the like. The memory 304 maycomprise, for example, random access memory (RAM), read only memory(ROM), a disk drive, an optical drive, a magnetic drive, and/or aUniversal Serial Bus (USB) drive. The module 305 for activatingactuators based on content analysis of the media may include circuitryand/or logic for performing special purpose functions relating to theoperation of a home gateway or AR server. The input/output devices 306may include, for example, a camera, a video camera, storage devices(including but not limited to, a tape drive, a floppy drive, a hard diskdrive or a compact disk drive), a receiver, a transmitter, a speaker, adisplay, a speech synthesizer, an output port, and a user input device(such as a keyboard, a keypad, a mouse, and the like), or a sensor.

Although only one processor element is shown, it should be noted thatthe computer may employ a plurality of processor elements. Furthermore,although only one computer is shown in the Figure, if the method(s) asdiscussed above is implemented in a distributed or parallel manner for aparticular illustrative example, i.e., the steps of the above method(s)or the entire method(s) are implemented across multiple or parallelcomputers, then the computer of this Figure is intended to representeach of those multiple computers. Furthermore, one or more hardwareprocessors can be utilized in supporting a virtualized or sharedcomputing environment. The virtualized computing environment may supportone or more virtual machines representing computers, servers, or othercomputing devices. In such virtualized virtual machines, hardwarecomponents such as hardware processors and computer-readable storagedevices may be virtualized or logically represented.

It should be noted that the present disclosure can be implemented insoftware and/or in a combination of software and hardware, e.g., usingapplication specific integrated circuits (ASIC), a programmable logicarray (PLA), including a field-programmable gate array (FPGA), or astate machine deployed on a hardware device, a computer or any otherhardware equivalents, e.g., computer readable instructions pertaining tothe method(s) discussed above can be used to configure a hardwareprocessor to perform the steps, functions and/or operations of the abovedisclosed method(s). In one example, instructions and data for thepresent module or process 305 for activating actuators based on contentanalysis of the media (e.g., a software program comprisingcomputer-executable instructions) can be loaded into memory 304 andexecuted by hardware processor element 302 to implement the steps,functions or operations as discussed above in connection with theexample method 200. Furthermore, when a hardware processor executesinstructions to perform “operations,” this could include the hardwareprocessor performing the operations directly and/or facilitating,directing, or cooperating with another hardware device or component(e.g., a co-processor and the like) to perform the operations.

The processor executing the computer readable or software instructionsrelating to the above described method(s) can be perceived as aprogrammed processor or a specialized processor. As such, the presentmodule 305 for activating actuators based on content analysis of themedia (including associated data structures) of the present disclosurecan be stored on a tangible or physical (broadly non-transitory)computer-readable storage device or medium, e.g., volatile memory,non-volatile memory, ROM memory, RAM memory, magnetic or optical drive,device or diskette and the like. More specifically, thecomputer-readable storage device may comprise any physical devices thatprovide the ability to store information such as data and/orinstructions to be accessed by a processor or a computing device such asa computer or an application server.

While various examples have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred example shouldnot be limited by any of the above-described example examples, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A method comprising: receiving, by a processingsystem deployed in a home network, a media stream from atelecommunications network, wherein the media stream is being deliveredto a user endpoint device in the home network connected to thetelecommunications network; identifying, by the processing system, ananchor in a scene of the media stream, wherein the anchor comprises apresence in the scene that has a physical effect on the scene;estimating, by the processing system, a type and a magnitude of thephysical effect of the anchor on the scene; identifying, by theprocessing system, an actuator that is capable of producing a physicaleffect in the real world to match the physical effect of the anchor onthe scene, wherein the actuator is integrated into a system in the homenetwork that comprises at least one of: a smart thermostat, a smartlighting system, or a smart speaker system; and sending, by theprocessing system, a signal to the system in the home network, andwherein the signal controls the actuator to produce the physical effectin the real world when the physical effect of the anchor on the sceneoccurs in the media stream.
 2. The method of claim 1, wherein the mediastream comprises a stream of a live event, and the identifying theanchor, the estimating, the identifying the actuator, and the sendingare performed as the media stream is being delivered to the userendpoint device.
 3. The method of claim 1, wherein the anchor is anobject that appears in the scene.
 4. The method of claim 1, wherein theanchor is an environmental effect of the scene on a camera that filmedthe scene.
 5. The method of claim 1, wherein the anchor is an event thatoccurs in the scene.
 6. The method of claim 1, wherein the estimating isperformed using a video processing technique to analyze a visualcomponent of the scene.
 7. The method of claim 1, wherein the estimatingis performed using an audio processing technique to analyze an audiocomponent of the scene.
 8. The method of claim 1, wherein the physicaleffect of the anchor on the scene comprises a force acting on orgenerated by the anchor.
 9. The method of claim 1, wherein the physicaleffect of the anchor on the scene comprises a change in a temperature ofthe scene that is caused by the anchor.
 10. The method of claim 1,wherein the actuator further comprises a haptic feedback device that isintegrated in the user endpoint device.
 11. The method of claim 1,wherein the actuator further comprises a haptic feedback device that isintegrated in a device located in physical proximity to the userendpoint device.
 12. The method of claim 1, wherein the identifying theactuator is based in part on a stored preference of a user of the userendpoint device.
 13. The method of claim 12, wherein the storedpreference relates to a medical condition of the user, and wherein atleast one of: a type or a magnitude of the physical effect in the realworld is selected to minimize aggravation of the medical condition. 14.The method of claim 12, wherein the stored preference relates to aninterest of the user, and wherein at least one of: a type or a magnitudeof the physical effect in the real world is selected to match theinterest.
 15. The method of claim 12, wherein the stored preferencerelates to a capability of the actuator, and wherein at least one of: atype or a magnitude of the physical effect in the real world is selectedto accommodate the capability.
 16. The method of claim 15, wherein thecapability of the actuator comprises at least one of: a current batterylife of the system in the home network into which the actuator isintegrated or a current connectivity strength of the system in the homenetwork into which the actuator is integrated.
 17. The method of claim1, further comprising: receiving, by the processing system, feedbackregarding a presentation of the physical effect in the real world to auser of the user endpoint device; and adjusting, by the processingsystem, a future iteration of the identifying the actuator based on thefeedback.
 18. The method of claim 1, wherein the anchor is renderedusing a special effect, and wherein the estimating is performed usingmetadata associated with the media stream.
 19. A non-transitorycomputer-readable medium storing instructions which, when executed by aprocessing system of a gateway deployed in a home network, cause theprocessing system to perform operations, the operations comprising:receiving a media stream from a telecommunications network, wherein themedia stream is being delivered to a user endpoint device in the homenetwork connected to the telecommunications network; identifying ananchor in a scene of the media stream, wherein the anchor comprises apresence in the scene that has a physical effect on the scene;estimating a type and a magnitude of the physical effect of the anchoron the scene; identifying an actuator that is capable of producing aphysical effect in the real world to match the physical effect of theanchor on the scene, wherein the actuator is integrated into a system inthe home network that comprises at least one of: a smart thermostat, asmart lighting system, or a smart speaker system; and sending a signalto the system in the home network, and wherein the signal controls theactuator to produce the physical effect in the real world when thephysical effect of the anchor on the scene occurs in the media stream.20. A device comprising: a processing system of a home network includingat least one processor; and a computer-readable medium storinginstructions which, when executed by the processing system, cause theprocessing system to perform operations, the operations comprising:receiving a media stream from a telecommunications network, wherein themedia stream is being delivered to a user endpoint device in the homenetwork connected to the telecommunications network; identifying ananchor in a scene of the media stream, wherein the anchor comprises apresence in the scene that has a physical effect on the scene;estimating a type and a magnitude of the physical effect of the anchoron the scene; identifying an actuator that is capable of producing aphysical effect in the real world to match the physical effect of theanchor on the scene, wherein the actuator is integrated into a system inthe home network that comprises at least one of: a smart thermostat, asmart lighting system, or a smart speaker system; and sending a signalto the system in the home network, and wherein the signal controls theactuator to produce the physical effect in the real world when thephysical effect of the anchor on the scene occurs in the media stream.